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Some  Consequences  of  the  Uncertainty  in 
IRT  Linking  Procedures 


Abstract 

In  many  practical  applications  of  item  response  theory,  the 
parameters  of  overlapping  subsets  of  test  items  are  estimated  from 
different  samples  of  examinees.  A  linking  procedure  is  then 
employed  to  place  the  resulting  item  parameter  estimates  onto  a 
common  scale.  It  is  standard  practice  to  ignore  the  uncertainty 
associated  with  the  linking  step  when  drawing  inferences  that 
involve  items  from  different  subsets,  a  situation  that  arises,  for 
example,  in  the  measurement  of  change.  This  paper  outlines  how 
the  uncertainty  can  be  accounted  for,  and  exemplifies  the  ideas 
with  a  jackknife  approximation  for  the  Stocking-Lord  linking 
procedure.  Examples  from  the  National  Assessment  of  Educational 
Progress  suggest  that  the  resulting  uncertainty  will  usually  be 
negligible  for  inferences  about  individuals,  but  can  constitute  a 
major  source  of  estimation  error  in  aggregate  statistics  such  as 
changes  in  group  means. 

Keywords:  Item  Response  Theory 

Linking  Transformations 

The  Stocking- Lord  Linking  Procedure 
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1.0  Introduction 


A  widely  cited  advantage  of  item  response  theory  (IRT)  in 
educational  measurement  is  its  capability  to  provide  proficiency 
estimates  on  a  common  scale  when  different  examinees  are 
administered  different  items,  or  when  examinees  are  administered 
different  items  at  different  points  in  time.  A  common  practice  is 
to  estimate  the  parameters  of  a  large  number  of  test  items,  treat 
the  estimates  as  known  true  parameters,  and  calculate  proficiency 
estimates  for  individuals  or  groups  based  on  responses  to  selected 
subsets  of  items.  Practical  considerations  often  preclude 
administering  all  items  to  a  single  sample  of  examinees  in  order 
to  obtain  the  initial  item  parameter  estimates;  rather,  estimates 
for  overlapping  sets  of  items  are  obtained  from  separate  samples 
of  examinees,  then  linked  to  a  common  scale.  While  it  is 
generally  recognized  that  the  parameters  of  the  required  linking 
functions  used  in  practice  are  estimates  rather  than  known 
constants,  the  effects  of  the  uncertainty  associated  with  them 
upon  subsequent  analyses  are  rarely  taken  into  account. 

This  paper  lays  out  a  framework  for  incorporating  the 
uncertainty  associated  with  IRT  linking  procedures  in  subsequent 
estimates  of  individual  or  group  change.  The  ideas  are 
implemented  for  the  linking  procedure  given  by  Stocking  and  Lord 
(1983),  and  illustrated  with  data  from  the  1084  and  1986  reading 
surveys  of  the  National  Assessment  of  Educational  Progress. 
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2.0  The  3 -Parameter  Logistic  Item  Response  Model 

The  3PL  model  expresses  the  probability  of  a  correct  response 

to  an  item  as  a  function  of  (i)  the  examinee's  proficiency  level 

0  ,  and  (ii)  three  parameters  characterizing  the  item, 

^.-(a. ,b. ,c . )  for  j— l,...,n.  The  parameter  a^.  ,  called  the 

discrimination  or  slope  parameter,  characterizes  the  item's 

sensitivity  to  proficiency.  The  parameter  bj ,  called  the 

threshold  parameter,  is  a  measure  of  item  difficulty.  The 

parameter  c^  is  the  probability  that  an  individual  with  very  low 

proficiency  will  respond  correctly  to  the  item.  The  conditional 

probability  of  a  correct  response  to  any  single  item,  denoted 

?.(&.),  is  obtained  as 

J  1 


P(x. ,~1|0. ,0.)  ■  P(x. .-110 . ,a. ,b. ,c.) 

ij  1  J  1  J  J  J 


-  C.  +  (1-Cj )/( l+exp[ -1 . 7a^ (0 . -b^ ) ] )  ,  (1) 

where  the  item  response  x..  =1  if  correct  and  0  if  not.  Under 

1J 

the  usual  assumption  of  local  or  conditional  independence,  the 
probability  of  a  vector  of  observed  item  responses,  x^  = 
(x^,,..x^n),  given  a  known  proficiency  value  6^,  can  be  expressed 
as  a  product  over  items  as  follows 


n  x. .  1-x. . 

p(x.|0.,/3)  =  n  1J(i-p(Xij-i|«if^))  1J 


u 


n 


(2) 


a  n  P .(6.) 

■  1  J  1 
J-l 


X.  . 

1J 


1-X.  . 


Because  P.(0.)  is  defined  as  a  function  of  a.(0.-b.),  the 
J  1  J  1  J 

origin  and  unit  of  measurement  of  the  proficiency  metric  are 
undetermined.  That  is,  for  any  rescaling  constants  A  and  B,  if 

_  I 

0.  =  A  0 .  +  B ,  b.  =  A  b .  +  B  and  a .  -A  a . , 

11  J  J  J  J 

then  a.  (0.  -b.  )-a.(0.  -b.)  and  P.(0.)  is  unchanged. 

J  1  J  J  1  j  J  i  6 

Since  any  such  linear  transformation  of  the  scale  retains  the 
meaning  and  the  implications  of  all  parameter  values,  the  unit- 
size  and  origin  of  the  0  scale  must  be  determined  arbitrarily  by 
the  researcher. 

Two  widely  used  procedures  for  estimating  the  item  parameters 
of  n  items  under  the  3PL  model  are:  joint  maximum 
likelihood,  the  approach  incorporated  in  the  LOGIST  program 
(Wingersky,  Barton,  and  Lord,  1982);  and  marginal  maximum 
likelihood,  the  approach  incorporated  in  the  BILOG  program 
(Mislevy  and  Bock,  1982).  In  both  of  these  programs,  the 
aforementioned  linear  indeterminacy  is  resolved  by  standardizing 
the  distribution  of  proficiency  in  the  calibration  sample  in  one 
way  or  another.  The  resulting  item  parameter  estimates,  and  the 
scale  they  implicitly  define,  are  then  typically  taken  as  fixed 
when  used  to  estimate  individual  examinees'  proficiencies  (as  may 
be  required  for  selection  or  placement  decisions)  or  population 
characteristics  such  as  group  means  (as  may  be  required  in 
educational  surveys  such  as  NAEP) .  In  order  to  focus  attention  on 
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the  impact  of  the  uncertainty  in  the  linking  functions,  we  shall 
not  deal  with  the  uncertainty  in  the  item  parameter  estimates 
themselves.  The  interested  reader  is  referred  to  Lewis  (1985)  and 
Tsutakawa  (1986)  for  more  on  this  latter  topic. 

3.0  Linking  Transformations 

Often,  it  is  not  feasible  to  administer  all  of  the  items  in  a 
large  item  pool  to  a  single  sample  of  examinees.  Instead, 
overlapping  subsets  of  items  are  administered  to  different  samples 
of  examinees.  When  practical  considerations  preclude  a  concurrent 
calibration  of  all  sample  data  together,  as  may  be  the  case  when 
the  various  samples  are  collected  at  different  points  in  time, 
then  independent  calibrations  must  be  performed  on  the  data 
collected  from  each  sample.  If  the  1RT  model  is  true,  the 
parameter  estimates  obtained  for  items  common  to  two  or  more 
calibrations  will  differ  by  (i)  estimation  error,  and  (ii)  an 
unknown  linear  transformation. 

In  this  paper,  we  address  the  simple  case  of  two  tests  that 
share  a  subset  of  common  items.  Each  test  is  independently 
calibrated  on  a  different  sample  of  examinees.  The  two 
calibration  samples  could  represent  the  same  group  of  examinees 
tested  at  two  different  points  in  time,  or  two  different  groups  of 
examinees  for  which  comparisons  are  to  be  made.  We  refer  to  the 
scale  established  by  the  calibration  of  the  first  sample  as  the 
target  scale  and  the  scale  established  by  the  calibration  of  the 
second  sample  as  the  provisional  scale.  The  inferential  problems 
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are,  first,  to  estimate  the  linear  transformation  needed  to  bring 
the  item  parameter  and  proficiency  estimates  from  the  provisional 
scale  to  the  target  scale,  and  second,  to  account  for  the 
uncertainty  of  the  linking  procedure  when  stating  the  precision  of 
resulting  statistics.  This  simple  case  can  be  generalized  to  the 
more  complex  calibration  problem  which  arises  when  multiple  forms 
of  a  test  are  calibrated  on  several  independent  samples  of 
examinees . 


3.1  The  Stocking- Lord  Linking  Procedure 

A  number  of  approaches  have  been  suggested  for  estimating 
linking  transformations.  Several  attempt  to  match  characteristics 
of  the  distributions  of  a  and  b  parameter  estimates  on  the  target 
scale  and  reexpressed  scale  (e.g.,  Marco,  1977),  possibly  with 
differential  weighting  of  estimates  to  account  for  the  precision 
with  which  they  have  been  estimated  (Linn,  Levine,  Hastings,  and 
Wordrop,  1980)  or  to  discount  the  influence  of  outliers  (Bejar  and 
Wingersky,  1981).  The  Stocking-Lord  (1983)  procedure,  which  we 
employ  in  the  sequel,  mi  limizes  the  average  squared  difference 
between  test  characteristic  curves  (TCCs)  estimated  from  the  two 
sets  of  item  parameters  available  for  the  common  items. 

The  input  data  to  the  Stocking-Lord  procedure  consists  of  two 
sets  of  parameter  estimates  for  the  common  items,  one  set 
expressed  on  the  target  scale  and  one  set  expressed  on  the 
provisional  scale.  For  item  j,  we  denote  these  estimated 

parameters  as  (a. and  (a.„  ,h.„  ,c.„  )  respectively. 

Jl  Jl  Jl  J  2p  J  2p  J  2p  K  y 
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The  goal  is  to  estimate  the  parameters  A  and  B  of  the  linking 

transformation  that  can  be  used  to  produce  rescaled  parameter 

estimates  (a.„  ,6.„  , c . .  ),  where 

J  2r  j  2r  j2r 


6..  =  A  6  .  „  +  B  ,  and 

j2r  J  2p 

c  =»  c 
j2r  C j 2p  ' 


(Note  that  the  estimate  of  the  lower  asymptote  parameter  c^p  is 
unaffected  bv  the  transformation.)  After  A  and  B  have  been 
estimated  from  the  items  common  to  both  calibrations,  this  same 
linking  transformation  is  applied  to  the  parameters  of  the  items 
that  appeared  in  the  second  calibration  only,  in  order  to  bring 
them  to  the  target  scale. 

Estimation  of  A  and  B  is  accomplished  by  minimizing  the 

squared  difference  between  estimated  true  scores  (expected  numbers 

correct)  on  the  n  common  items  at  N  preselected  values  of  0.  The 
c 

function  to  be  minimized  is 


2 

f(A,B,0)  =  1/N  £  ((-.(1,0,0.)  -  f,(A ,B,0.))  (3) 

.  ,  i  1  1 

1  =  1 

where  f ^ ( 1 , 0 , 0  .  )  is  the  true  score  associated  with  the  proficiency 
level  8^,  calculated  from  the  common  items  using  the  item 
parameter  estimates  expressed  on  the  target  scale,  and  ^(A.B,?^) 
is  the  true  score  associated  with  the  proficiency  level  0 
calculated  from  the  common  items  using  the  item  parameter 
estimates  which  were  originally  obtained  on  the  provisional  scale 
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and  then  reexpressed  on  the  target  scale  with  the  rescaling 
parameters  A  and  B.  That  is, 


^(1,0,0.) 


(l-c.jl)/{l+exp[-1.7ajl(*i-6.1)]  ) 


and 


cJ2p  a  (l-£J2p)/IUexp[-K7A'1a.2p(»..<A6j2pAB))! 


J-l 


(1'£j2r)/(1+exp^1,7“j2r(*i'6j2r)  ]  1 


The  values  6=(6, . <?,)  ,  which  are  selected  rather  than 

1  N 

estimated,  play  the  role  of  the  independent  variables  in  a 
regression  analysis.  They  should  be  selected  to  insure  that  the 
equation  given  in  (3)  is  minimized  over  the  entire  (expected) 
range  of  the  target  proficiency  scale. 

We  note  in  passing  that  under  this  procedure,  the  common 
items  end  up  with  three  sets  of  item  parameter  estimates,  one  set 
expressed  on  the  provisional  scale,  and  two  sets  expressed  on  the 
target  scale.  Alternative  procedures  for  combining  the  two  sets 
of  estimates  expressed  on  the  target  scale  are  given  in  McKinley 
(1988)  . 
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3.2  A  Jackknife  Approximation  for  the  Uncertainty  of  the 

Stocking-Lord  Linking  Procedure 

The  uncertainty  associated  with  the  estimated  rescaling 
parameters  A  and  B  of  the  Stocking- Lord  linking  procedure  can  be 
approximated  using  a  Jackknife  procedure  (Mosteller  and  Tukey, 
1977).  Although  alternative  Jackknife  implementations  may  be 
appropriate  for  the  problem  described  here,  for  the  purposes  of 
illustration,  we  present  a  single  variation  only.  The  variation 
presented  is  an  example  of  an  interpenetrating  Jackknife 
procedure.  It  consists  of  three  steps.  First,  the  set  of 
common  items  used  to  define  the  transformation  are  divided  into 
ten  equal  length  subsets  with  approximately  equal  average 
difficulty.  Second,  the  function  given  in  (3)  is  minimized  ten 
times.  Each  minimization  is  accomplished  using  all  but  one  of  the 
item  subsets  defined  in  step  1.  Finally,  the  observed  variation 
among  the  A  and  B  parameter  estimates  obtained  from  the  ten 
minimizations  is  used  to  estimate  a  covariance  matrix  which 
quantifies  uncertainty  due  to  (i)  the  imprecision  of  the  estimated 
item  parameters,  and  (ii)  lack  of  fit  from  the  IRT  model.  This 
procedure  is  illustrated  with  data  from  the  National  Assessment  of 
Educational  Progress  in  Section  5. 

The  jackknife  procedure  described  above  measures  variation 
arising  from  two  sources:  estimation  error  and  model  misfit.  The 
uncertainty  associated  with  estimation  error  can  often  be 
decreased  by  increasing  the  size  of  the  calibration  samples.  To 
decrease  the  uncertainty  associated  with  model  misfit,  it  is  also 
necessary  to  have  a  large  number  of  linking  items.  To  see  this, 
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note  that,  If  the  IRT  model  were  correct,  the  differences  between 
sets  of  (a,b,c)  estimates  obtained  from  different  increasingly 
large  ramples  of  examinees  would  be  accounted  for  totally  by  a 
linear  transformation.  In  this  case,  consistent  estimates  of  the 
linking  parameters  could  be  obtained  with  as  few  as  two  linking 
items.  When  the  IRT  model  does  not  fit,  however,  different  sets 
of  linking  items  will  tend  to  provide  different  estimates  of  the 
linking  parameters  even  as  calibration  sample  sizes  increase 
without  bound.  In  this  latter  case,  it  is  clear  that  the  model 
misfit  component  of  uncertainty  can  only  be  reduced  by  increasing 
the  number  of  linking  items.  Moreover,  the  linking  items  should 
be  chosen  so  as  to  be  representative  of  the  set  of  all  items  which 
might  have  been  used  to  estimate  the  linking  function. 

4.  How  the  Uncertainty  in  Linking  Procedures  Propagates  to 

Subsequent  Analyses 

In  this  section,  we  show  how  the  uncertainty  associated  with 
an  IRT  linking  procedure  can  be  accounted  for,  in  the  context  of 
measuring  change.  As  before,  we  consider  the  simple  case  of  onlv 
two  tests  sharing  a  single  subset  of  common  items  The  first  test 
is  administered  to  a  group  of  examinees  at  time  1.  The  second 
test  is  administered  to  the  same  group  of  examinees  at  time  2. 

Our  primary  interest  is  to  measure  the  change  in  proficiency 
observed  over  time  for  individual  examinees  and  for  specified 
population  subgroups.  We  assume  that  a  covariance  matrix 
quantifying  the  uncertainty  associated  with  the  parameters  of  the 
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linear  transformation  used  to  link  the  two  tests  has  been 


estimated  (as  with  a  jackknife  approximation,  for  example) . 

We  first  consider  the  problem  of  estimating  the  change  in 
proficiency  for  a  single  examinee.  Let  denote  a  proficiency 

estimate  calculated  for  the  ith  examinee  at  time  1  using  the 
estimated  item  parameters  which  were  originally  obtained  on  the 
target  scale.  Let  0.^  denote  a  proficiency  estimate  calculated 
for  the  same  examinee  at  time  2  using  the  estimated  item 
parameters  which  were  originally  expressed  on  the  provisional 
scale.  And  finally,  let  0  denote  a  proficiency  estimate 
obtained  for  the  same  examinee  at  time  2  using  the  item  parameters 
which  were  originally  estimated  on  the  provisional  scale  and 
subsequently  reexpressed  on  the  target  scale;  that  is,  ^p2r  ” 

A  A  A 

A  L,  +  B.  Since  8.,  and  8.~  are  both  expressed  on  the  target 
i2p  ll  i2r 

scale,  an  estimate  of  the  change  in  proficiency  for  this  examinee 

can  be  obtained  from  the  difference,  D.  =  8.„  -  9...  If  the 

l  i2r  ll 

parameters  of  the  linking  transformation  were  known  without  error, 
then  the  standard  error  of  this  estimated  change  would  be  given  by 


SE(Di>  ’  SE<»i2r  -*il>  ‘  '“'iZr  * 


(4) 


where  a._  and  o.,  are  the  standard  errors  of  the  proficiency 
i2r  ll 

estimates  8 . „  and  9.,,  respectively.  (As  is  usually  the  case,  we 
i2r  ll 

have  also  assumed  independent  errors  across  tests.) 

Now  a  will  be  a  function  of  the  item  parameters  which  were 
originally  estimated  on  the  target  scale,  whereas  o ^  will  be  a 
function  of  the  item  parameters  which  were  originally  estimated  on 


the  provisional  scale  and  then  reexpressed  on  the  target  scale. 

Thus,  any  procedure  which  accounts  for  the  uncertainty  of  the 

transformation  used  to  link  the  two  tests  will  affect  the 

calculation  of  a ..  but  not  a.,.  To  calculate  a,.  ,  note  that 
i2r  ll  i2r 

A  A  A  A  A 

9.„  -  A  6. .  +  B,  and  that  the  estimated  standard  error  of  9..  , 
i2r  i2p  i2p 

denoted  a^2p’  can  calculated  as  a  function  of  item  parameters 
which  have  not  yet  been  rescaled  and  are  thus  unaffected  by  the 
uncertainty  of  the  linking  procedure. 

A  A  A 

As  a  first  step,  define  a  covariance  matrix  for  (0.^  >  A,  B] 
as  follows: 


i2p 


0 

2 

7  A 

7ab 


AB 

2 

7  B 


where  <7.,  cr  ,  and  aA„  quantify  estimation  variation  for  the 
A  B  AB  ^  y 

parameters  A  and  B  of  the  linking  transformation.  The  quantities 
cta>  0^,  and  o  can  be  approximated  using  the  jackknife  procedure 
given  in  the  previous  section.  Second,  note  that 


Var(fli2r)  =  Var(A*i2p  +  B) 


Var(g(«i2p,A,B)) 


d(z)  ,  d(e.) 

,  3(e) 

S 

Slgi  ,  aigi 

,  IU 1 

86..  5A 

i2p 

3B 

89..  3A 

i2p 

3B 

A  .  «12p  ■  1  1  S  I  A  .  «12p  .  1 
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5.  A  Numerical  Illustration 

In  this  section,  data  available  from  the  National  Assessment 
of  Educational  Progress  (NAEP) ,  a  congressionally  mandated  survey 
of  the  educational  achievement  of  American  students,  is  used  to 
approximate  the  uncertainty  of  the  Stocking-Lord  linking  procedure 
and  to  evaluate  the  consequences  of  that  uncertainty.  Data  from 
two  NAEP  surveys  are  used:  the  1984  Reading  Survey  and  the  1986 
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Reading  Survey.  Both  of  these  surveys  were  independently  scaled 
using  a  three  parameter  logistic  IRT  model.  Item  parameters  were 
estimated  using  BILOG  (Mislevy  &  Bock,  1982)  and  mean 
proficiencies  for  population  subgroups  were  obtained  using  the 
plausible  values  methodology  given  in  Mislevy  and  Sheehan  (1987)  . 
These  data  are  used  to  illustrate  the  consequences  of  the 
uncertainty  of  the  transformation  parameter  estimates  from  the 
Stocking-Lord  linking  procedure.  Because  NAEP  data  support 
inferences  about  aggregate  statistics  such  as  group  means  but  not 
about  individuals'  proficiencies,  we  use  real  NAEP  data  to 
demonstrate  procedures  for  changes  in  group  means  but  simulated 
data  for  changes  in  individual  prof iciencies . 

5.1  The  NAEP  Data 

Mean  reading  proficiencies  for  the  three  age  groups  which 
were  assessed  by  NAEP  in  1984  and  1986  are  given  in  Table  1.  The 
first  row  of  the  table  provides  1984  age  group  means  expressed  on 
the  1984  calibration  scale.  For  the  purpose  of  this  illustration, 
the  1984  calibration  scale  is  designated  as  the  target  scale.  The 
second  and  third  rows  of  the  table  provide  1986  age  group  means 
expressed  on  the  provisional  scale  (the  1986  calibration  scale) 
and  the  target  scale  (the  1984  calibration  scale).  The  Stocking- 
Lord  linking  procedure  was  used  to  estimate  the  linear 
transformation  needed  to  express  the  1986  means  on  the  1984 
calibration  scale.  The  table  also  provides  estimated  standard 


errors  for  each  mean. 


Table  1  about  here 


5.2  Quantifying  the  Uncertainty  of  the  NAEP  Link 

The  1984  NAEP  survey  contained  128  cognitive  reading  items. 
The  1986  NAEP  survey  contained  107  cognitive  reading  items,  76 
which  were  common  to  the  1984  assessment  and  31  which  were 
administered  for  the  first  time  in  1986.  The  linking 
transformation  needed  to  express  the  item  parameters  obtained  from 
the  calibration  of  the  1986  data  on  the  scale  established  by  the 
calibration  of  the  1984  data  was  estimated  using  the  Stocking-Lord 
linking  procedure,  as  implemented  in  the  TBLT  computer  program 
(Stocking,  1986).  The  generally  satisfactory  results  can  be  seen 
in  Figure  1,  which  shows  the  TCCs  of  the  first  and  second 
calibrations  of  the  common  items  after  reexpression,  and  in  Figure 
2,  which  plots  the  b-parameter  estimates  from  the  first  and 
reexpressed  second  calibrations.  The  jackknife  procedure 
described  in  Section  3  was  used  to  approximate  the  uncertainty 
associated  with  the  estimated  parameters  of  the  linking 
transformation.  The  results  are  given  in  Table  2. 


Figures  1  and  2  and  Table  2  about  here 
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5.3  Inference  for  a  Single  Examinee 


The  artificial  data  set  constructed  for  this  analysis 
contained  simulated  responses  for  five  examinees  to  two  tests. 

The  first  test  consisted  of  30  items  selected  from  the  1984  NAEP 
reading  survey.  The  second  test  consisted  of  30  items  selected 
from  the  1986  NAEP  reading  survey,  half  of  which  were  common  to 
the  1984  survey.  For  a  given  examinee,  responses  were  generated 
in  accordance  with  the  3PL,  with  item  parameter  estimates  for  the 
first  test  taken  from  the  1984  NAEP  calibration  run  and  item 
parameter  estimates  for  the  second  test  taken  from  the  1986  NAEP 
calibration  run.  So  that  the  proficiency  of  a  given  simulee  was 
the  same  on  both  tests,  a  value  of  8  was  specified  for  the  first 
test  and  (0-B)/A  was  used  for  the  second.  Simulees'  8  values  on 
the  first  test  were  -1.0,  -0.5,  0.0,  0.5,  and  1.0.  The  response 
vectors  generated  according  to  these  specifications  are  given  in 
Table  3. 


Table  3  about  here 


Treating  the  item  parameter  estimates  as  known,  maximum 
likelihood  estimates  (MLEs)  of  8  and  associated  standard  errors 
were  obtained  for  each  response  pattern  using  the  BILOG  program. 
They  are  shown  in  Table  4,  with  the  values  for  the  second  test 
shown  before  and  after  reexpression.  Table  5  provides  estimated 
standard  errors  for  the  change  from  the  first  test  to  the  second 
using  (4),  which  does  not  take  the  uncertainty  of  A  and  B  into 


account,  and  (6),  which  does.  The  increase  in  standard  errors  is 
negligible,  about  2-percent  on  the  average.  An  approximate 
variance  components  analysis  is  given  in  Table  6.  For  each 
response  pattern  considered,  the  total  error  variance  is  estimated 
using  (6)  which  includes  components  due  to  both  sampling  and 
linking.  The  contribution  due  to  sampling  alone  is  estimated 
using  (4)  and  the  contribution  due  to  linking  is  obtained  by 
subtraction.  The  table  shows  that  for  each  response  pattern 
considered,  the  relative  increase  in  uncertainty  is  negligible, 
accounting  for  about  three  percent  of  the  total  error  variance  on 
the  average. 


Tables  4,  5  and  6  about  here 


5.4  Inference  for  Group  Means 

The  changes  in  the  mean  reading  proficiencies  of  students 
aged  9,  13  and  17,  over  the  two  year  period  from  1984  to  1986,  as 
estimated  from  the  NAEP  data,  are  given  in  Table  7.^  The  table 
also  provides  approximate  standard  errors  calculated  using  (4)  and 
(6).  Whereas  the  size  of  standard  errors  increased  by  only  about 
2-percent  for  estimates  of  change  of  individuals .  the  increase  in 
standard  errors  for  groups  is  about  200-percent!  An  approximate 

^  These  figures  are  shown  for  illustrative  purposes  only,  and 
are  not  to  be  taken  as  estimates  of  changes  in  reading  proficiency 
during  the  period  due  to  certain  anomalies  in  the  1985/86  NAEP 
data.  The  interested  reader  is  referred  to  Beaton  (1988)  for 
further  information. 
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variance  components  analysis  is  given  in  Table  8.  The  table  shows 
that  the  component  due  to  linking  represents  approximately  90- 
percent  of  the  total  error  variance,  on  the  average.  To  put  these 
results  in  another  perspective,  the  change  in  mean  reading 
proficiency  at  each  age  level  is  expressed  in  standard  error  units 
in  Table  9.  The  table  shows,  for  example,  that  the  decrease  in 
the  mean  reading  proficiency  of  9  year  olds  is  approximately  three 
standard  errors  when  the  uncertainty  of  the  linking  procedure  is 
not  accounted  for,  but  only  one  standard  error  when  it  is. 


Tables  7,  8  and  9  about  here 


6.0  Summary 

A  common  problem  in  applied  work  with  item  response  theory  is 
to  express  item  parameter  estimates  from  separate  calibrations  on 
the  same  scale,  based  on  the  multiple  estimates  for  subsets  of 
items  common  to  two  or  more  calibrations.  Several  methods  have 
been  proposed  for  estimating  the  optimal  linear  transformations 
for  this  purpose,  including  the  Stocking-Lord  (1983)  procedure  for 
matching  test  characteristic  curves.  After  the  resulting 
transformations  have  been  applied,  the  uncertainty  associated  with 
them  is  rarely  taken  into  account  in  subsequent  analyses  of 
individual  or  group  levels  of  proficiency. 

This  uncertainty  can  be  expressed  in  terms  of  a  covariance 
matrix  of  estimation  errors,  which  can  be  approximated  empirically 
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through  a  procedure  such  as  the  jackknife.  With  an  approximation 
of  the  sampling  covariance  matrix  of  estimation  errors  of  the 
parameters  of  a  linking  transformation,  one  can  readily  derive 
standard  errors  for  change  scores  or  comparisons  that  take  this 
additional  uncertainty  into  account. 

Using  data  from  the  1984  and  1986  reading  surveys  of  the 
National  Assessment  of  Educational  Progress,  this  paper  used  the 
jackknife  to  approximate  the  uncertainty  of  the  linking 
transformation  between  the  two  assessments.  Its  effect  was  found 
to  be  negligible  in  the  context  of  drawing  inferences  about  change 
of  individuals,  since  its  magnitude  was  much  smaller  than  the 
uncertainty  arising  from  having  only  the  limited  numbers  of  item 
responses  from  individuals  that  generally  characterize  individual 
testing  programs.  Correct  standard  errors  were  only  about  2- 
percent  larger  than  those  that  ignored  linking  uncertainty.  The 
effect  was  substantial  in  the  context  of  estimating  group  changes, 
however,  leading  to  correct  standard  errors  that  were  200-percent 
larger.  The  differential  impact  is  due  to  the  fact  that  sampling 
variances  of  group  means  are  much  smaller  than  sampling  variances 
of  individual  scores,  while  the  sampling  variance  of  the  linking 
transformation  is  the  same  in  both  cases. 
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Table  1 


Mean  Proficiencies 

Estimated  from  the  1984  and  1986  NAEP  Reading  Surveys 
With  Standard  Errors  in  Parentheses 


Year 

Scale 

Ace  9 

Aee  13 

Age  17 

84 

84  Calib. 

-0. 7  5  2 ( .020) 

0 . 150 ( . 014) 

0 . 766 ( . 018) 

86 

86  Calib. 

-0 . 375( . 025) 

0 . 571( . 019) 

0.874( .018) 

86 

84  Calib. 

-0. 864( .028) 

0 . 198 ( . 022) 

0.538( .020) 

The  1984  sample  included  over  22,000  students  at  each  age 
level.  The  1986  sample  included  approximately  7,000  Age  9 
students,  6,000  Age  13  students,  and  16,000  Age  17  students. 
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Table  2 


Results  of  the  Jackknife  Approximation 


for  the 

Stocking - 

Lord  Linking 

Procedure 

„  1 

Run 

I  terns 

A 

B 

0 

76 

1.122196 

-0.442910 

1 

68 

1.118018 

-0.449670 

2 

68 

1.126296 

-0.447837 

3 

68 

1.121856 

-0.449472 

4 

68 

1.110982 

-0.433893 

5 

68 

1.114703 

-0.426793 

6 

68 

1.128065 

-0.430320 

7 

69 

1.125834 

-0.446748 

8 

69 

1 . 128753 

-0.440663 

9 

69 

1.112862 

-0.447648 

10 

69 

1.135424 

-0.455858 

Jackknife 

Parameter 

Estimate 

2 

a  A 

0.00512 

2 

a  B 

0.00740 

aAB 

-0.00238 

The  parameter  estimates,  A  and  B,  obtained  from  Run  0  were  used 
to  reexpress  the  1986  results  on  the  1984  scale.  The  parameter 
estimates  obtained  from  Runs  1  through  10  were  used  only  to 
estimate  the  uncertainty  of  the  linking  procedure. 
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Table  3 


Simulated  Responses  To  Test  1 
Administered  at  Time  1 


Generating 

Value 


1.0 

11000 

11000 

10011 

00101 

00000 

01010 

0.5 

00110 

10101 

10000 

10011 

00111 

11001 

0.0 

00010 

11101 

11100 

00100 

OHIO 

11100 

0.5 

11111 

01111 

11111 

00111 

01101 

11111 

1.0 

11111 

11111 

11111 

01111 

10110 

11111 

Simulated  Responses  To  Test  2 
Administered  at  Time  2 


Generating 

Value 


-.50 

00010 

01000 

00011 

11000 

10000 

00001 

-  .05 

11001 

01000 

01011 

11101 

01100 

11000 

0.39 

01100 

01101 

10011 

00111 

11111 

10100 

0.84 

00011 

11111 

10111 

11110 

11101 

01111 

1.29 

11111 

11111 

11111 

10111 

10110 

OHIO 

Table  4 

Maximum  Likelihood  Estimates  of  Reading  Proficiency 
At  Time  1  and  Time  2 
For  Five  Simulated  Subjects 
With  Estimated  Standard  Errors  in  Parentheses 


Value  Estimated  at  Time  2 


Generating 

Value 

-1.0 

-0.5 

0.0 

0.5 

1.0 


Value 
Estimated 
at  Time  1 
-1.062  (.625) 
-0.662  (.489) 
-0.502  (.470) 
0.748  (.546) 
1.177  (.662) 


Before 

Reexorersion 
-0.375  (.422) 
-0.116  (.534) 
0.249  (.360) 
0.824  (.409) 
1.434  (.512) 


After 

Reexpression 
-0.864  (.474) 
-0.574  (.560) 
-0.163  (.404) 
0.482  (.459) 
1.172  (.574) 
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Table  5 


An  Estimate  of  the  Change  in  Reading  Proficiency 
From  Time  1  to  Time  2 
For  Five  Simulated  Subjects  ^ 

With  Approximate  Standard  Errors 


Change  in 
Generating 
Values 
0 
0 
0 
0 
0 


Estimated 

Change 

0.198 

0.088 

0.339 

-0.266 

-0.005 


S.E. 

Method  1 
0.784 
0.743 
0.620 
0.713 
0.876 


S.E. 

Method  2 
0.790 
0.779 
0.625 
0.718 
0.883 


Method  1  refers  to  the  method  which  assumes  that  the 
linking  function  is  known  without  error,  as  in  equation  (4); 

Method  2  refers  to  the  method  which  accounts  for  the 
uncertainty  of  the  linking  procedure  as  in  equation  (6) . 


Table  6 

A  Comparison  of  Approximate  Variance  Components 
For  Inferences  About  Change  at  the  Individual  Level 


Generating 

Value 

-1.0 

-0.5 

0.0 

0.5 

1.0 


Totai. 
Variance 
.6241 
.6068 
.3906 
.5155 
.  7797 


Component 
Due  to 
Sampling 
.6146 
.5520 
.  3844 
.5084 
.7674 


Component 
Due  to 
Linking 
.0094 
.0548 
.0062 
.0071 
.0123 


Linking 
Variance 
as  %  of 
Total 
Variance 

1.5 
9.0 

1.6 
1.4 
1.6 
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Table  7 


An  Estimate  of  the  Change  in  Mean  Reading  Proficiency 
From  1984  to  1986  ^ 

With  Approximate  Standard  Errors 


Estimated 

S.E. 

S.E. 

Aze 

Chanee 

Method  1 

Method  2 

9 

-0.112 

.034 

.105 

13 

0.048 

.026 

.084 

17 

-0.228 

.027 

.066 

Metnod  1  refers  to  the  method  which  assumes  that  the 
linking  function  is  known  without  error,  as  in  equation  (4); 
Method  2  refers  to  the  method  which  accounts  for  the 
uncertainty  of  the  linking  procedure  as  in  equation  (6) . 


Table  8 

A  Comparison  of  Approximate  Variance  Components 
For  Inferences  About  Change  at  the  Group  Level 


Linking 

Variance 


Component 

Component 

as  %  of 

Total  ^ 

due  to 

due  to 

Total 

Age 

Variance 

SamDl inz 

Linkine 

Variance 

9 

.01  in 

.0012 

.0098 

89.5 

13 

.0071 

.0007 

.0064 

90.1 

17 

.0044 

.0007 

.0037 

84.1 

1  Total  Variance  refers  to  the  estimated  variance  of  the  change 
in  mean  reading  proficiency  from  1984  to  1986. 
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Table  9 


The  Estimated  Change  in  Mean  Reading  Proficiency 
from  1984  to  1986 
Expressed  in  Standard  Error  Units 


Method  1  Method  2 


Age 

S.E. 

Units 

S.E.  Units 

9 

-3. 

.29 

-1.07 

13 

1 

.85 

0.57 

17 

-8 

.44 

-3.45 
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Figure  2 

Comparison  of  Item  b  Parameter  Estimates 
Reexpressed  1986  Estimates  vs.  1984  Estimates 
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Dr .  R .  Darrell  8ock 
University  of  Chicago 
NORC 

6030  South  Ellis 
Chicago,  IL  60637 

Cdt.  Arnold  Bohrer 

Sectie  Psycho  I og i sch  Onderzoek 

Rekruter i ngs-En  Se I ec t i ecentrum 

Kwartier  Koningen  Astrid 

Bru i jnstraat 

1120  Brussels,  BELGIUM 

Dr.  Robert  Breaux 
Code  7B 

Naval  Trai.ning  Systems  Center 
Orlando,  FL  32313-7100 

Dr.  Robert  Brennan 
American  College  Testing 
Proqrams 
P.  0.  Box  168 
Iowa  City,  IA  52243 

Dr.  James  Carlson 
American  College  Testing 
Program 
P.0.  Box  168 
Iowa  Ci ty,  IA  52243 

Dr  .  John  B .  Car ro I  I 
409  Elliott  Rd. ,  North 
Chapel  Hill.  NC  27514 

Dr.  Robert  M.  Carroll 
Chief  of  Naval  Operations 
0P-01 B2 

Washington,  DC  20350 

Dr.  Raymond  E.  Christal 
UES  LAMP  Science  Advisor 
APHRL/MOEL 

Brooks  AFB,  fX  /8235 

Dr .  Norman  Cl  iff 
Department  of  Psychology 
Univ.  of  So.  California 
Los  Angeles,  CA  90039-1061 


1988/04/20 


Educational  Testing  Service /M i s I evy 


D i rec  to  r , 

Manpower  Support  and 
Readiness  Program 
Center  for  Naval  Analysis 
2000  North  Beauregard  Street 
Alexandria,  VA  22311 

Dr .  Stan  ley  Co  I  I  ye  r 
Office  of  Naval  Technology 
Code  222 

800  N.  Quincy  Street 
Arlington,  VA  22217-5000 

Dr.  Hans  F.  Crombag 
Faculty  of  Law 
University  of  Limburg 
P.0.  Box  616 
Maastr i cht 

The  NETHERLANDS  6200  MD 

Or.  Timothy  Oavey 
Educational  Testing  Service 
Princeton,  NJ  03541 

Dr.  C.  M.  Oavton 
Department  of  Measurement 
Statistics  &  Evaluation 
College  of  Educat i on 
University  of  Maryland 
College  Park,  MD  20742 

Or.  Ralph  J.  DeAyala 
Measurement,  Statistics, 
and  Evaluation 
Benjamin  Bldg.,  Rm.  4112 
University  of  Maryland 
College  Park,  MD  20742 

Or.  Dattprasad  Divgi 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr.  Hei-Ki  Dong 
Bel!  Communications  Research 
6  Corporate  Place 
PYA-1K226 

Piscataway,  NJ  08854 


Dr.  Fritz  Drasgow 
University  of  Illinois 
Department  of  Psychology 
603  E .  Dan i e I  St . 

Champaign,  IL  61820 

Defense  Technical 

Information  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  22314 
Attn:  T  C 
( 12  Cop i es> 

Dr.  Stephen  Dunbar 
224B  Lindquist  Center 
for  Measurement 
University  of  I owa 
Iowa  City,  IA  52242 

Dr.  James  A.  Earles 

Air  Force  Human  Resources  Lab 

Brooks  AF8,  TX  78235 

Dr.  Kent  Eaton 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria.  VA  22333 

Dr.  John  M.  Eddins 
University  of  Illinois 
252  Engineering  Research 
Laboratory 

103  South  Mathews  Street 
Urbana,  IL  61801 

Dr.  Susan  Embretson 
University  of  Kansas 
Psychology  Department 
426  Fraser 
Lawrence,  KS  66045 

Dr.  George  Englehard,  Jr. 
Division  of  Educational  Studies 
Emory  University 
210  Fishburne  Bldg. 

Atlanta,  GA  30322 

Dr.  Benjamin  A.  Fairbank 
Performance  Metrics,  Inc. 

5825  Ca I  I aghan 
Suite  225 

San  Antonio,  TX  78228 


1986/04/20 


Educational  Testing  Service/Mi slevy 


Or.  P-A.  Federico 

Code  51 

NPRDC 

San  Diego,  CA  92152-6800 

Or .  Leonard  Fe I dt 
Lindquist  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  IA  52242 

Dr.  Richard  L.  Ferguson 
American  College  Testing 
P . 0 .  Box  168 
Iowa  C i  ty  ,  IA  52243 

Dr.  Gerhard  F i scher 
L i eb i ggasse  5/3 
A  1010  Vienna 
AUSTRIA 

Dr.  Myron  Fischl 
U.S.  Army  Headquarters 
OAPE-MRR 
The  Pentagon 

Washington,  DC  20310-0300 

Prof'.  Donald  Fitzgerald 
University  of  New  England 
Department  of  Psychology 
Armidale,  New  South  Wales  2351 
AUSTRALIA 

Mr .  Pau I  Foley 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 

Dr.  Alfred  R.  Freglv 
AFOSR/NL,  Bldg.  410 
Bolling  AF8 ,  OC  20332-6443 

Dr.  Robert  D.  Gibbons 
Illinois  State  Psychiatric  Inst. 
Rm  529W 

1601  W.  Taylor  Street 
Chicago,  IL  60612 

Or.  Janice  Gi fford 
University  of  Massachusetts 
School  of  Education 
Amherst,  MA  01003 


Dr .  Robert  G I ase r 
Learning  Research 

&  Development  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 

Dr.  Bert  Green 
Johns  Hopkins  University 
Department  of  Psychology 
Charles  &  34th  Street 
Baltimore,  MD  21218 

DORNIER  GMBH 

P.0.  Box  1420 

D-7990  Fr i edr i chshal en  1 

WEST  GERMANY 

Dr.  Ronald  K.  Hamb I e ton 
University  of  Massachusetts 
Laboratory  of  Psychometric 
and  Evaluative  Research 
Hills  South ,  Room  152 
Amherst,  MA  01003 

Dr .  Del wyn  Harn i sch 
University  of  Illinois 
51  Gerty  Drive 
Champaign,  IL  G1320 

Dr.  Grant  Henning 
Senior  Research  Scientist 
Division  of  Measurement 
Research  and  Services 
Educational  Testing  Service 
Princeton,  NJ  08541 

Ms.  Rebecca  Hetter 
Navy  Personnel  R&D  Center 
Code  63 

San  Diego,  CA  92152-6800 

Dr ,  Pau I  W .  Holland 
Educational  Testing  Service,  2 1  —  T 
Rosedale  Road 
Princeton,  NJ  08541 

Prof.  Lutz  F.  Hornke 
Institut  fur  Psychologie 
PWTH  Aachen 
Jaegerstrasse  17/19 
D-5100  Aachen 
WEST  GERMANY 


1988/04/20 


Educational  Testing  Serv i ce/M i s I evy 


Dr.  Paul  Horst 
677  G  Street,  #134 
Chula  Vista,  CA  92010 

Mr.  Dick  Hoshaw 
OP-135 

Ar I i ngton  Annex 
Room  2834 

Washington,  DC  20350 

Dr.  Lloyd  Humphreys 
University  of  Illinois 
Department  of  Psychology 
603  East  Daniel  Street 
Champaign,  IL  61820 

Dr.  Steven  Hunka 
3-104  Educ.  N. 

University  of  Alberta 
Edmonton,  Alberta 
CANADA  T6G  2G5 

Dr.  Huynh  Huynh 
Co  I  I  ege  of  Educati on 
Univ.  of  South  Carolina 
Columbia,  SC  29200 

Dr.  Robert  Jannarone 
Elec,  and  Computer  Eng.  Dept. 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr .  Douglas  H .  Jones 
Thatcher  Jones  Associates 
P.0.  Box  6640 
10  T raf  a  I  gar  Court 
Lawrencev i I  I e,  NJ  08648 

Dr.  Milton  3.  Katz 
European  Science  Coordination 
Office 

U.S.  Army  Research  Institute 
Box  65 

FPO  New  York  09510-1500 

Prof.  John  A.  Keats 
Department  of  Psychology 
University  of  Newcastle 
N.S.W.  2308 
AUSTRALIA 


Dr.  G.  Gage  Kingsbury 

Portland  Public  Schools 

Research  and  Evaluation  Department 

501  North  Dixon  Street 

P.  0.  Box  3107 

Portland,  OR  97209-3107 

Dr .  William  Koch 
Box  7246,  Meas.  and  Eval.  Ctr. 
University  of  Texas-Austin 
Austin,  TX  78703 

Dr.  James  Kraatz 
Computer-based  Education 
Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Leonard  Kroeker 
Navy  Personnel  R&D  Center 
Code  62 

San  Diego,  CA  92152-6800 

Dr.  Jerry  Lehnus 

Defense  Manpower  Data  Center 

Su i te  400 

1600  Wi I  son  Bl vd 

Rosslyn,  VA  22209 

Dr.  Thomas  Leonard 
University  of  Wisconsin 
Department  of  Statistics 
1210  West  Dayton  Street 
Madison,  WI  53705 

Dr .  Michael  Levine 
Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 

Dr .  Charles  Lew i s 
Educational  Testing  Service 
Princeton,  NJ  08541-0001 

Dr.  Robert  L.  Linn 
Campus  Box  249 
University  of  Colorado 
Boulder,  CO  80309-0249 


1988/04/20 


Educational  Testing  Serv i ce/Mi s I evy 


Or.  Robert  Lockman 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16263 
Alexandria.  VA  22302-0268 

Or.  Frederic  M.  Lord 
Educational  Testing  Service 
Princeton,  NJ  03541 

Dr.  George  B.  Macreadv 
Department  of  Measurement 
Statistics  S  Evaluation 
College  of  Education 
University  of  Maryland 
College  Park,  MD  20742 

Dr.  Gary  Marco 
Stop  31-E 

Educational  Testing  Service 
Pr  i  no  e  ton  ,  N.J  08451 

Dr.  james  R.  McBride 
The  Psychological  Corporation 
1250  Sixth  Avenue 
San  D i ego ,  CA  92101 

Or.  Clarence  C.  McCormick 
HQ,  USMEPCOM/MEPCT 
2500  Green  Bay  Road 
North  Chicago.  It  60064 

Dr.  Robert  McKinley 
Educational  Testing  Service 
1 6— T 

Princeton,  NJ  08541 

Dr.  James  McMicnael 
Technical  Director 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  8arbara  Means 
SRI  international 
333  Ravenswood  Avenue 
Menlo  Park,  CA  94025 

Or.  Robert  Mislevy 
Educational  Testing  Service 
Princeton,  NJ  08541 


Dr .  W i  I  I  i am  Montague 

NPRDC  Code  13 

San  Diego,  CA  92152-6800 

Ms.  Kathleen  Moreno 
Navy  Personnel  R&D  Center 
Code  62 

San  Diego,  CA  92152-6800 

Headguarters  Marine  Corps 
Code  MPI-20 
Washington,  DC  203S0 

Dr.  W.  Alan  Nicewander 
University  of  Oklahoma 
Department  of  Psychology 
Norman,  OK  73071 

Deputy  Technical  Director 

NPRDC  Code  OlA 

San  Diego,  CA  92152-6800 

Director,  Training  Laboratory, 
NPRDC  (Code  05> 

San  Diego,  CA  92152-6800 

Director.  Manpower  and  Personnel 
laboratory , 

NPRDC  (Code  06) 

San  Diego,  CA  92152-6800 

Director,  Human  Factors 

&  Organizational  Systems  Lab, 
NPRDC  (Code  07) 

San  Diego,  CA  92152-6800 

Library.  NPRDC 
Code  P201L 

San  Diego,  CA  92152-6800 

Commanding  Officer, 

Navn I  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

Dr.  Harold  F .  O’Neil,  Jr. 

School  of  Education  -  WPH  801 
Department  of  Educational 
Psychology  &  Technology 
University  of  Southern  California 
Los  Angeles.  CA  90089  0031 


1988/04/20 


Educational  Testing  Serv i ce/Mi s I evy 


Dr .  James  B.  Olsen 
W IC A T  Sv5 terns 
1875  Sou  tii  State  Street 
Orem,  UT  84053 

0 '  t  i  oo  c  t  Nava!  Research, 

Code  1M2CS 
800  N.  Quincy  Street 
Arlington,  VA  22217-5000 
(6  Copies' 

Office  of  Naval  Research, 

Code  125 

800  N.  Quincy  Street 
Arl ington,  VA  22217-5000 

Assistant  for  MPT  Research, 
Development  and  Studies 
OP  0187 

Washington.  DC  20370 

Dr.  Judith  LVasanu 
Basic  Research  Office 
Army  Research  Institute 
5001  Eisenhower  Avenue 
A  '  e  <3iidr  i  a ,  VA  22333 

Dr.  Jesse  Or  I  a  risky 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  St. 
Alexandria,  VA  2231 1 

Dr .  Rando I ph  Park 
Army  Research  Institute 
5001  Eisenhower  Blvd. 

Alexandria,  VA  22333 

Wavne  M.  Pat i ence 
American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Dupont  Circle,  NW 
Washington,  DC  20036 

Dr .  James  Pau I  son 
Department  of  Psychology 
Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

Dept,  of  Administrative  Sciences 
Code  54 

Naval  Postgraduate  School 
Monterey,  CA  93943-5026 


Department  of  Operations  Research, 
Naval  Postgraduate  School 
Monterey,  CA  93940 

Dr.  Mark  D.  Pe;kase 
ACT 

P.  0.  Box  168 
I owa  City,  J  A  52243 

Dr.  Malcolm  Ree 

afhrl/moa 

Brooks  APB,  TX  78235 

Dr.  Barry  Riegelhaupt 
HumRRO 

1100  South  Washington  Street 
Alexandria,  VA  22314 

Dr .  Carl  Ross 
CNET-PDCD 
Bui  I d i ng  90 

Great  Lakes  NFC,  IL  60083 
Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Fumiko  Samejima 
Department  of  Psychology 
University  of  Tennessee 
310B  Austin  Peay  Bldg. 

Knoxv  i  I  I e.  IN  37316-0900 

Mr.  Drew  Sands 

NPRDC  Code  62 

San  Diego,  CA  92152-6800 

Lowe  I  I  Schoer 

Psychological  &  Quantitative 
Foundati ons 
College  of  Education 
University  of  Iowa 
Iowa  City,  I A  52242 

Dr .  Mary  Schra  tz 

Navy  Personnel  R&D  Center 

San  Diego.  CA  92152-6800 

Dr .  Dan  Segal  I 

Navy  Personnel  P&D  Center 

San  Diego,  CA  92152 


1988/04/20 


Educational  Testing  Serv i ce/Mi s  I  e vy 


Dr.  W.  Steve  Sell  man 
OAS0 (MRA&L) 

2B269  The  Pentagon 
Washington,  DC  20301 

Or.  Kazuo  Shigemasu 
7-9-24  Kugenuma~Ka i gan 
Fu j i sawa  251 
JAPAN 

Dr .  W i  I  I  i am  Sims 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria.  V A  22302-0268 

Dr.  H.  Wallace  Sinai ko 
Manpower  Research 

and  Advisorv  Services 
Smithsonian  Institution 
801  North  Pitt  Street,  Suite  120 
Alexandria,  VA  22314-1713 

Dr .  Richard  E .  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

Or.  Richard  C.  Sorensen 
Navy  Personnel  RSD  Center 
San  Diego,  CA  92152-6800 

Dr.  Paul  Speckman 
University  of  Missouri 
Department  of  Statistics 
Co  I umb i a ,  MQ  65201 

Dr.  Judy  Spray 
ACT 

P.0.  Box  168 
[ owa  C i ty ,  IA  52243 

Or.  Martha  Stocking 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr ,  W i  I  I  i am  Stout 
University  of  Illinois 
Department  of  Statistics 
101  I  I  I  ini  Hall 
725  South  Wright  St. 

Champaign,  IL  61820 


Dr.  Hariharan  Swaminathan 
Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

Mr.  Brad  Sympson 

Navy  Personnel  R&D  Center 

Code-62 

San  Diego,  CA  92152-6800 

Dr.  John  Tangnev 

AFOSR/NL ,  Bldg.  410 

Bo  I  I i ng  AFB,  DC  20332-6448 

Dr.  Kikumi  Tatsuoka 
CERL 

252  Engineering  Research 
Laboratory 

103  S.  Mathews  Avenue 
U  r  b  a  n  a ,  I L  61301 

Dr.  Maurice  Tatsuoka 
220  Educat ion  Bldg 
1310  S .  Sixth  St . 

Champaign,  IL  61820 

Dr.  Javid  Thissen 
Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

Mr.  Gary  Thomasson 
University  of  Illinois 
Educational  Psychology 
Champaign,  IL  61820 

Dr.  Robert  Tsutakawa 
University  of  Missouri 
Department  of  Statistics 
222  Math.  Sciences  Bldg, 
Columbia,  MO  8521 1 

Dr.  Ledyard  Tucker 
University  of  Illinois 
Department  of  Psychology 
803  E.  Daniel  Street 
Champ  .  i gn ,  IL  61820 


1988/04/20 


Educational  Testing  Serv i ce/M i s I evy 


Or.  Vern  W .  IJrry 
Personnel  R&O  Center 
Office  of  Personnel  Management 
1900  E.  Street,  NW 
Washington,  DC  20415 

Dr.  David  Vale 
Assessment  Systems  Corp. 

2233  University  Avenue 
Suite  440 

St.  Pau I .  MN  551 14 

Dr.  Frank  1  .  V i c i no 
Navv  Personnel  R&O  Center 
San  Diego,  CA  92152-6800 

Dr .  Howard  Wa i ner 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  M i n g - M e i  Wang 
Lindquist  Center 
for  Measu  remen  t 
On i vers i ty  of  Iowa 
i ova  C i ty ,  I A  52242 

Or.  Thomas  A.  Warm 
Coast  Guard  Institute 
P.  0.  Substation  18 
Oklahoma  City,  OK  73169 

Dr.  Brian  Waters 
HumRRO 

12908  Argy I e  Circle 
Alexandria,  VA  22314 

Dr.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455-0344 

Dr.  Ronald  A.  Weitzman 
Box  146 

Carme I ,  CA  93921 

Major  John  Welsh 

AFHRL/MOAN 

Brooks  AF8,  TX  78223 


Dr.  Douglas  Wetzel 
Code  51 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6300 

Dr .  Rand  R .  Wilcox 
University  of  Southern 
Cal  i  f orn i a 

Depa'tment  of  Psychology 
Los  Angeles,  CA  90089-1061 

German  Military  Representative 
ATTN:  Wolfgang  Wildgrube 
St  re i tkraef teamt 
D-5300  Bonn  2 

4000  Brandywine  Street,  NW 
Washington,  DC  20016 

Dr.  Bruce  Willi ams 
Department  of  Educational 
Psychology 

University  of  Illinois 
Urbana,  IL  61801 

Dr.  Hilda  Wing 
NR C  MH- 176 

2101  Constitution  Ave. 
Washington,  DC  20418 

Dr .  Martin  F .  W i skof f 
Defense  Manpower  Data  Center 
550  Camino  El  Estero 
Suite  200 

Monterey,  CA  93943-3231 

Mr.  John  H.  Wolfe 

Navy  Personnel  R&D  Center 

San  Dieqo,  CA  92152-6800 

Dr.  George  Wong 
Biostatistics  Laboratory 
Memorial  S I oan-Kette r i ng 
Cancer  Center 
1275  York  Avenue 
New  York,  NY  10021 

Dr.  Wallace  Wulfeck,  III 
Navy  Personnel  R&D  Center 
Code  51 

San  Diego,  CA  92152-6800 


1988/04/20 


Educational  Testing  Serv i ce/Mi s I evy 


Or.  Kentaro  Yamamoto 
03-T 

Educational  Testing  Service 
Rosedale  Road 
Princeton,  NJ  08541 

Dr.  Wendy  Yen 
CTB/McGraw  Hi  I  1 
Oel  Monte  Research  Park 
Monterey,  CA  93940 

Dr.  Joseph  L.  Young 
National  Science  Foundation 
Poom  320 

1800  G  Street,  N.W. 
Washington,  DC  20550 

Mr.  Anthony  R.  Zara 
National  Council  of  State 
Boards  of  Nursing,  Inc. 
625  North  Michigan  Avenue 
Suite  1544 
Chicago,  IL  60611 

Dr.  Peter  Stoloff 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 


